Summary

• Reduce memory usage of batch flat KNN when the final projection does not include the vector column (Closes perf: High memory usage for batched vector query on flat KNN #6940).
• Heap candidates no longer retain whole scan RecordBatches (and the full vector column) in the common columns=[...] case.
• When the vector column is projected, retain only a per-row copy of the vector (not the entire scan batch).

Implementation notes

• Scanner::flat_knn computes retain_vector from the final projection and plumbs it into KnnBatchParams.
• KNNVectorDistanceExec batch mode uses a two-tier candidate representation:
  • RowIdOnly { query_index, distance, row_id } when retain_vector=false
  • WithVector { ..., slim_batch, row_index, vector_row } when retain_vector=true

Memory (unit test)

Synthetic workload matching test_batch_knn_heap_memory: num_rows=4096, dim=512, m=10, k=10.

What is measured: sum of Arrow get_array_memory_size() over candidate heap entries (not peak RSS or end-to-end query memory).

Notes:

• Baseline simulates pre-fix behavior: each of m×k candidates holds a shallow clone of the full scan batch (vector + _rowid for all num_rows). In this synthetic setup the same batch is cloned repeatedly, so the counter sums shared buffers per candidate (worst-case retention model).
• WithVector is on the order of m×k×d×4 (= 204,800 bytes for vector data) plus one deduplicated slim batch (_rowid only).
• Indexed batch KNN and other large columns are out of scope for this PR.

Latency (local, secondary)

Release build (uv run maturin develop --uv --release), use_index=false, with_row_id=true.

Workload: rows=200_000, dim=256, m=10, k=10 (medians over 8 rounds, 2 warmup).

Batch latency improved ~1.71× vs baseline in this workload; separate-query path unchanged.

Test plan

• cargo test -p lance --lib batch_knn
• cargo clippy -p lance --tests -- -D warnings
• cd python && uv run pytest python/tests/test_vector_index.py -k batch